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(57) Abstract: A super-resolution enhanced 
image generating system is described for 
generating a super-resolution -enhanced image 
from an image of a scene, identified as image 
go, comprising a base image and at least 
one other image gj, the system comprising 
an initial super-resolution enhanced image 
generator, an image projector module and 
a super-resolution enhanced image estimate 
update generator module. The initial 
super-resolution enhanced image generator 
module is configured to use the image go to 
generate a super-resolution enhanced image 
estimate. The image projector module is 
configured to selectively use a warping, 
a blurring and/or a decimation operator 
associated with the image gj to generate a 
projected super-resolution enhanced image 
estimate. The super-resolution enhanced 
image estimate update generator module is 
configured to use the input image and the 
super-resolution enhanced image estimate 
to generate an updated super-resolution 
enhanced image estimate. 
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System And Method For Providing Multi-Sensor Super^Resolution 

Field Of The Invention 

The invention is generally related to the field of image processing of images, and more 
particularly to systems for enhancing the resolution of images. The invention is specifically directed 
to a system and method of generating an enhanced super-resolution image of a scene from a plurality 
of input images of the scene that may have been recorded with different image recording sensors. 

Background Of The Invention 

An image of a scene as recorded by camera are recorded at a particular resolution, which 
depends on the particular image recording medium or sensor that is used to record the image. Using 
one of a number of so-called "super resolution enhancement" methodologies, an image of enhanced 
resolution can be generated from a plurality of input images of the scene. A number of super- 
resolution enhancement methodologies assume that the images are recorded by the same sensor. 
However, in many applications, images of a scene will typically be recorded using different image 
recording sensors. For example, a color image of a scene is recorded by an image recording medium 
that actually comprises three different sensors, one sensor for each of the red, blue and green primary 
colors. A problem arises in such applications, since conventional super-resolution-enhancement 
methodologies can only be applied on multiple images generated by a similar sensor. 

Summary Of The Invention 

The invention provides a new and improved system and method of generating a super- 
enhanced resolution image of a scene from a plurality of input images of a scene that may have been 
recorded with different image recording sensors. 

In brief summary, the invention provides a super-resolution enhanced image generating 
system for generating a super-resolution-enhanced image from an image of a scene, identified as 
image comprising a base image and at least one other image g^, the system comprising an initial 
super-resolution enhanced image generator, an image projector module and a super-resolution 
enhanced image estimate update generator module. The initial super-resolution enhanced image 
generator module is configured to use the image to generate a super-resolution enhanced image 
estimate. The image projector module is configured to selectively use a warping, a blurring and/or 
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a decimation operator associated with the image gf to generate a projected super-resolution enhanced 
image estimate. The super-resolution enhanced image estimate update generator module is 
configured to use the input image and the super-resolution enhanced image estimate to generate 
an updated super-resolution enhanced image estimate. 

Brief Description Of The Drawings 

This invention is pointed out with particularity in the appended claims. The above and 
further advantages of this invention may be better understood by referring to the following 
description taken in conjimction with the accompanying drawings, in which: 

FIG. 1 is a functional block diagram of a super-resolution enhanced image generating system 
for generating a super-enhanced resolution image of a scene firom a plurality of input images of a 
scene that may have been recorded with different image recording sensors, constructed in accordance 
with the invention; 

FIG. 2 is a flow chart depicting operations performed by the super-resolution enhanced image 
generating system depicted in FIG. 1 ; 

FIG. 3 is a flow chart depicting operations performed by the super-resolution enhanced image 
generating system in connection with a demosaicing operation. 

Detailed Description of an Illustrative Embodiment 

FIG. 1 is a functional block diagram of a super-resolution enhanced image generating system 
10 for generating a super-enhanced resolution image of a scene from a plurality of input images of 
a scene that may have been recorded with different image recording sensors, constructed in 
accordance with the invention. With reference to FIG. 1, the super-resolution enhanced image 
generating system 10 includes an initial super-resolution enhanced image estimate generator 11 , an 
super-resolution enhanced image estimate store 12, an image projector 13, an input image selector 
14, a virtual input image generator 1 5, an error image generator 1 6, and an super-resolution enhanced 
image update generator 17, all under control of a control module 18. Under control of the control 
module 1 8, the super-resolution enhanced image estimate generator 1 1 receives a base input image 
**^^" of a scene and generates an initial estimate for the super-resolution enhanced image to be 
generated, which it stores in the super-resolution enhanced image estimate store 12. 
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Also under control of the control module 1 8, the image selector selects one of a plurality of 
other input images {gi}^^^ of the scene, which may have been recorded by another sensor. 

Thereafter, the control module 18 enables the image projector 13 to generate a projected image of 
the super-resolution enhanced image estimate image in the super-resolution enhanced image estimate 
store, in this case using the imaging model for the image recording sensor that had been used to 
record the selected input image g,. The projected super-resolution enhanced image estimate and the 
selected input image are both coupled to the virtual input image generator 15. The virtual input 
image generator 15 receives the projected super-resolution enhanced image estimate as generated 
by the image projector 13 arid the selected input image g^, i^ 1^ and iperforms two operations. First, 
for each pixel in corresponding locations in the projected image estimate and the selected input 
image, the virtual input image generator 15 generates a normalized cross-correlation value in a 
neighborhood of the respective pixel, and determines whether the normalized cross-correlation value 
is above a selected threshold value. The normalized cross-correlation value is an indication of the 
aflHne similarity between the intensity of the projected image and the intensity of the selected input 
image, that is, the degree to which a feature, such as an edge, that appears near the particular location 
in the projected image images is also near the same location in the selected input image. If the 
normalized cross-correlation value is above the threshold, the virtual input image generator generates 
affine parameters using a least-squares estimate for the affme similarity in the window, and uses 
those affme parameters, along with the pixel value (which represents the intensity) at the location 
in the selected input image to generate a pixel value for the location in the virtual input image. On 
the other hand, if the normalized cross-correction value is below the threshold, the virtual input 
image generator 15 assigns the location in the virtual input imaige the pixel value at tha:t location in 
the projected image. The virtual input image generator 15 performs these operations for each of the 
pixels in the projected image as generated by the image projector 13 and the selected input image 
g,., i^ 1 , thereby to generate the virtual input image. 

The error image generator 16 receives the projected image as generated by the image 
projector 13 and the virtual input image as generated by the virtual input image generator 15 and 
generates an error image that reflects the difference in pixel values as between the two images. The 
super-resolution enhanced image estimate update generator 1 7 receives the error image as generated 
by the error image generator 16 and the super-resolution enhanced image estimate store 12 and 
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generates an updated super-resolution enhanced image estimate, which is stored in the super- 
resolution enhanced image estimate store 12. 

If there is another input image {g^, ^ , after the updated super-resolution enhanced image 

estimate has been stored in the store 12, the control module can enable the various modules 13-17 
to repeat the operations described above in connection with another input image. These operations 
can be repeated for each of the input images. After all, or a selected subset, of the input images 

{^i}/.] have been selected and used as described above, the control module 18 can terminate 

processing, at which point the image in the super-resolution enhanced image estimate store 12 will 
comprise the super-resolution enhanced image. Alternatively, or in addition, the control module 1 8 

can enable the operations in connection with the input images {g| }^_^ > or the selected subset, 

through one or more subsequent iterations to further enhance the resolution of the image in the super- 
resolution enhanced image estimate store 12. 

Details of the operations performed by the initial super-resolution enhanced image estimate 
generator 1 1 , the image projector 1 3, "the virtual input image generator 1 6, the error image generator 
16 and the super-resolution enhanced image estimate update generator 17 will be described below. 

By way of background, a super-resolution enhanced image of a scene is typically generated 

from a set (g/l^.j of "n" input images of a scene, all of which have been recorded by the same 

image recording sensor. The input images are indexed using index "i." It will be assumed that all 
of the images are the result of imaging a high-resolution image /of a scene, which high-resolution 
image initially is unknown. The imaging model, which models the various recorded images g^^ 
usually includes one or more of a plurality of components, the components including a geometric 
warp, camera blur, decimation, and added noise. The goal is to find a high-resolution image/that, 
when imaged into the lattice of the input images according to the imaging model, provides a good 

prediction for the low-resolution images g,. If is an estimate of the unknown image / a prediction 
Qvror image "J/* for the estimate f is given by 

^, = &-B,(w,{f))i (1) 
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where " W is an operator that represents geometric warping as between the "i-th" input image g, and 
the estimate / of the unknown image/, "B," is an operator that represents the blurring as between 

the "i-th" input image g, and the estimate / of the unknown image/, and " i " represents decimation, 

that is, the reduction in resolution of the estimate / of the unknown image /to the resolution of the 

"i-th" input image g,. The blurring operator "B;' represents all of the possible sources of blurring, 
including blurring that may be due to the camera's optical properties, blurring that may be due to the 
camera's image recording sensor, and blurring that may be due to motion of the camera and/or 
objects in the scene while the camera's shutter is open. The warping operator " ^F," represents the 
displacement or motion of points in the scene as between the respective "i-th" input image "g," and 
the estimate / . An example of the motion that is represented by the warping function " FF/' is the 
displacement of a projection of a point in the scene from a pixel having coordinates (x^yy{) in an 
image to a pixel having coordinates (Xi',y;) in the estimate / . It will be appreciated that the 

information defining the respective input images are in the form of arrays, as are various operators. 
It will also be appreciated that the arrays comprising the various operators will generally be different 
for each of the "i" input images g^. Typically, super-resolution image generating systems determine 

the estimate / of the unknown image /by minimizing the norm of the prediction error image rf, 

(equation (1)) as among all of the input images g^. 

The super-resolution image generating systems described in connection with equation (1) . 

generate a super-resolution enhanced image from a plurality of input images j^^^ , all of which 

were recorded by the same image recording sensor. The super-resolution enhanced image generating 
system in accordance with the invention generates a super-enhanced image from a set of input 
images, which need not have been recorded using the same image recording sensor. More 

specifically, the super-resolution enhanced image generating system generates an estimate / of a 
high-resolution version of a base image "go" of a scene using that image "g^' and a set ^ of 

other input images of the scene. The base image g^ and the other input images {gf}" may, but 



<WO 03047a34A2_l_> 



wo 03/047234 



PCT/IB02/05437 



-6- 

need not, have been recorded using different image recording sensors; that is, no assumption is made 
that the image recording sensors that were used to record the input images {g, Y. ^ are similar to 

each other, or to the sensor that was used to record the base image "g^" The super-resolution 
enhanced image generating system 10 can generate an initial estimate from the base image using 
any of a number of methodologies as will be apparent to those skilled in the art; one methodology 
that is useful in connection with one particular application will be described below. 

The super-resolution enhanced image generating system makes use of an observation that 
aligned images as recorded by different image recording sensors are often statistically related. The 
super-resolution enhanced image generating system makes use of a statistical model **Af * that relates 

a projected image p. = Bf{w^{^f^ i to the respective input image '*g" as recorded by the image 
recording sensor. In that case, a virtual input image can be defined as 

= M{p,,g,) (2) 

For the projected image p. = B^{fV^(^f )) * ^® current estimate of the super-resolution 

enhanced image, and, as described above, "5^," "PF" and "I" represent the blurring, warping and 
decimation operators, respectively, for the respective "i-th" input image, i^ 1 . 

After the respective projected image and the virtual input image g. for a particular input 

image gf have been generated, a virtual prediction error image "v/'xan be generated as 

V, = gi ' Pi (3) 

Thereafter, the super-resolution enhanced image generating system uses the virtual prediction error 
image " v," from equation (3) in generating the super-resolution enhanced image in a manner similar 
to that described above in connection with equation (1), but using the virtual prediction error image 
V,. instead of the prediction error image 

The super-resolution enhanced image generating system makes use of a statistical model M 

that is based on a local photometric afiBne alignment as between the intensities of the input images | g. | ^ ^ 
and the intensities of the projected images. In particular, the super-resolution enhanced image 
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generating system makes use of small corresponding pixel neighborhoods in the projected image />, 
and the associated input image g, and generates estimates for an affine transformation relating the 
intensity values of the image p, to the intensity values of the respective input image g„ that is 

Pi{^>y) = o{x,y) gi(^,y) + where (x,y) represents the location of the respective pixel. 

(In the following discussion, the use of "(x,y)" may be dropped for clarity.) The super-resolution 
enhanced image generating system then uses the mapping to generate the respective virtual input 
image as, 

iXx^y) = a{x,y)gXx,y) + b{x,y) (4). 

and the prediction error image v, as described above in connection with equation (3). Under the 
assimiption that the input image is contaminated with zero-mean white noise, which typically is the 
case, the optimal estimation for the affine relation between each projected image and the respective 
input image g/ provides coefficients "a" and "b" such as to minimize 



where "u" and "v" define a small neighborhood R around each of the respective pixels (x,y). 

The super-resolution enhanced image generating system generates the pixel value (intensity) 

for a pixel at location (x,y) in the virtual input image = , ) (reference equation (3)) as 

follows. Initially, the super-resolution enhanced image generating system generates an estimate 
"c(x,y)" of a centralized and normalized cross-correlation in a weighted window of size "2k+l " by 
"2k+l," for a selected value of "k," around the respective pixel at location (x,y) as follows: 



(6). 



where Pi(x,y)y g,(^,>'). Pi g,{x,y)> Pi^{xyy),and g,^{x,y) are weighted averages 
around the pixel at coordinates (x,y) in the respective image org^. That is, for example. 
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PiK^^y) = 



•ksu.vsk 

^ v)pi(x+u,y+ v)g,. (x+u,y+v) 

-kiu.vSk 



(7), 



and so forth, where "w(1u, v/* is a weighting kernel. In one embodiment, the values selected for the 
weighting kemel decrease with increasing absolute values of "u" and "v," so that less weight will 
be given in the weighted simis to values of pixels in the window with increasing distance from the 
pixel at locatipn (x,y), thereby to provide for a reduction in spacial discontinuities in the afline 
parameters "a" and "b" (equation (4)). 

After the super-resolution enhanced image generating system has generated the cross- 
correlation estimate c(x,y) of the cross-correlation for a pixel at location (x,y), it compares the 
absolute value of the cross-correlation estimate c(x,y) to a threshold value "T." If the super- 
resolution enhanced image generating system determines that the absolute value of the cross- 
correlation estimate c(x,y) is less than the threshold value 'T," the value of the pixel at location (x,y) 
in the virtual input image is set to correspond to the value of the pixel at location (x,y) in the 



projected image/?,., that is, ^y(x,>;)= pX^.y)^ 



On the other hand, if the super-resolution enhanced image generating system determines that 
the absolute value of the estimate c(x,y) is greater than or equal to the threshold value "r," it 
' generates coefficients "a" and "b" for use in generating the least-squares estimate for the affine 
similarity in the window aroimd the pixel at location (x,y) as follows 



a 




b 





Pi{xyy) 1 



-I r 



Pi 8,{x,y) 



(8) 



and then generates the pixel value for the pixel in location (x,y) of the virtual input image as 

gii^yy) = ag,{x,y)-\-b (9), 

(compare equation (4)) where "a" and "b" in equation (9) are as determined from equation (8). The 
super-resolution enhanced image generating system performs the operations described above in 
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connection with equations (6) through (9) for each of the pixels that are to comprise part of the 
respective virtual input image g, After the super-resolution enhanced image generating system has 

completed the respective virtual input image g , it generates the value for the virtual prediction error 

image v, using equation (3). After the super-resolution enhanced image generating system has 
generated the virtual prediction error image v„ it will use the virtual prediction error image to update 
the estimate / of the unknown image/ The super-resolution enhanced image generating system 

can repeat these operations for each of the input images {g^r}J_i update the estimate / . In 

addition, the super-resolution enhanced image generating system can repeat the operations with the 
some or all of the input images {g. }"^^ , and possibly also the base input image g^, through one or 

more subsequent iterations to further update the super-resolution enhanced image estimate f . 

FIG. 2 depicts a flow chart outlining operations that are performed by the super-resolution 
enhanced image generating system 10 as described above in connection with FIG. 1 and equations 
(2) through (9) in connection with generating and updating the a super-resolution enhanced image 

estimate / . The operations will be apparent to those skilled in the art with reference to the above 

discussion, and so the various steps outlined in the flow chart will not be described in detail herein 
In one embodiment, the super-resolution enhanced image generating system 1 0 is used in 
connection with demosaicing images that are associated with respective color channels of a CCD 
(charge coupled device) array that may be used in, for example a digital camera. Typically, a CCD 
array has a plurality of individual light sensor elements that are arranged in a plurality of rows and 
columns, with each light sensor element detecting the intensity of light of a particular color incident 
over the respective element's surface. Each sensor element is associated with a pixel in the image, 
and identifies the degree of brightness of the color associated with that pixel. The sensor element 
performs the operation of integrating the intensity of light incident over the sensor element's area 
over a particular period of time, as described above. 

For a CCD array that is to be used in connection with the visual portion of the 
electromagnetic spectrum, each two-by-two block of four light sensor elements comprises light 
sensor elements that are to sense light of all three primary colors, namely, green red and blue. In 
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each two-by-two block, two light sensor elements are provided to sense light of color green, one light 
sensor element is provided to sense light of color red and one light sensor element is provided to 
sense light of color blue. Typically, the two light sensor elements that are to be used to sense light 
of color green are along a diagonal of the two-by-two block and the other two light sensor elements 
are along the diagonal orthogonal thereto. In a typical CCD array, the green light sensor elements 
are at the upper right and lower left of the block, the red light sensor element is at the upper left and 
the blue light sensor element is at the lower right. It will be appreciated that the collection of sensor 
elements of each color provides an image of the respective color, with the additional proviso that the 
two green sensor elements in the respective blocks define two separate input images. In that case, 
if, for example, the CCD array has "m" rows-by-"n" columns, with a total resolution of m*n (where 
represents multiplication), the image that is provided by the respective sets of green sensors in 
the respective blocks will have a resolution of V4*m*n, and the image that is provided by each of the 
red and blue sensors will have a resolution of %*m*n. In the demosaicing operation, the super- 
resolution enhanced image generating system generates a super-resolution enhanced image 
associated with each color channel, using information fi*om the image that is associated with the 
respective color channel as well as the images that are associated with the other colors as input 
images. In this connection, the collection of light sensor elements of the CCD array that is associated 
with each of the colors is a respective "image recording sensor" for purposes of the super-resolution 
enhanced image generating system, so that there will be four image recording sensors and four input 
images, with one input image being associated with each of the image recording sensors. The super- 
resolution enhanced image that is generated by the super-resolution enhanced image generating 
system for each of the color channels will have a resolution of m-by-n. The super-resolution 
enhanced images generated for the various color channels by the super-resolution enhanced image 
generating arrangement can then be used in generating a composite color image of higher resolution 
than the original image provided by the CCD array. 

In the demosaicing operation, the super-resolution enhanced image generating system 
actually generates four super-resolution enhanced images, one super-resolution enhanced image for 
each of the red and blue color channels, and two for the green color channel. For each super- 
resolution enhanced image, and with reference to the discussion above in connection with equations 
3 through 1 2, the base input image go is the image provided by the collection of light sensor elements 
associated with the respective color channel (with the two green light sensors in the respective blocks 

representing different color channels), and the other input images } .^j (in this case, n=3) will 
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be provided by the collection of light sensor elements associated with the other three color channels, 
with the proviso, noted above, that the sensor elements in the respective blocks that are associated 
with the green color define separate channels for the demosaicing operation. For example, for the 
green channels, the base images are the images that are provided by the green light sensor 
elements of the respective positions in the blocks comprising the CCD array, and the red and blue 

input images {^/} will be provided by the collection of light sensor elements in the respective 

blocks that are associated with the red and blue channels. Similarly with base image and other input 
images associated with each of the other color channels. 

After the super-resolution enhanced image generating system receives the four input images, 
it first uses the input images associated with the respective color channels to generate, for each color 

channel, an initial estimate / of the super-resolution enhanced image /for the respective color 

channel. Accordingly, the super-resolution enhanced image generating system uses the input image 
that is associated with the one of the green channels as the base image from which it generates the 

initial estimate / of the super-resolution enhanced image /that is associated with that green color 

channel, and similarly for the other color channels. In one embodiment, for each of the red and blue 
color channels, as well as the other green color channel, the super-resolution enhanced image 
generating system uses a linear interpolation as between points for the respective color channel in 
the array to provide initial pixel values for the array positions that are not provided by the CCD 
array. As mentioned above, the CCD array, in each two-by-two block of light sensor elements, one 

red light sensor element is provided at the upper left of the block. For the initial estimate / for the 

super-resolution enhanced image that is associated with the red channel, the red light sensor element 
in each block will provide the red channel pixel value that is associated with that pixel in the image 
that is associated with the red light sensor element. The super-resolution enhanced image generating 
system will generate red channel pixel values for the pixels that are associated with other three points 
in the block for which red light sensor elements are not provided by interpolating between the pixel 
values provided by red light sensor elements in neighboring blocks. The super-resolution enhanced 
image generating system generates the blue channel pixel values and the other blue channel pixel 
values for the pixels that are associated with the three points in the block for which blue light sensor 
elements are not provided in a similar manner. 
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As noted above, two green light sensor elements are provided in each block, one at the upper 
right and one at the lower left. In one embodiment, the super-resolution enhanced image generating 
system also uses a linear interpolation for the points in the block for which green light sensor 
elements are not provided. However, in this case, the super-resolution enhanced image generating 
system will select one of the green light sensor elements in each block, and perform a linear 
interpolation between pixel values provided by the selected green light sensor element and one of 
the green light sensor elements in adjacent blocks. For example, if the super-resolution enhanced 
image generating system makes use of the green light sensor element at the upper right of a block, 
in generating the pixel value for the point associated with the upper left of the same block, the super- 
resolution enhanced image generating system makes use of the pixel value generated by the green 
light sensor elements of that block and of the block to the left. In that case, in generating the pixel 
value for the point associated with the lower right of the same block, the super-resolution enhanced 
image generating system makes use of the pixel value generated by the green light sensor elements 
of that block and of the adjacent block down the array. In that embodiment, the super-resolution 
enhanced image generating system will not use the pixel value generated by the other green light 
sensor element in the block, in this example, the green light sensor element at the lower left comer 

of the block. Generating the initial estimate f for the green chaimel in this maimer results in a 

smooth initial estimate with satisfactory properties in connection with convergence to a result 
associated with the green color channel having desirable characteristics. On the other hand, if pixel 
values provided by both of the green light sensor elements in a block were to be used, the result 
associated with the green color channel may maintain some of the artifacts froni the initial estimate 

After the super-resolution enhanced image generating system has generated the initial 
estimate f for the respective color channels, it will project the initial estimate f that had been 

generated for the respective color channel back onto the input image for that color channel, thereby 
to generate the projected estimate image for the respective color channel. The super-resolution 

enhanced image generating system performs that operation, by convolving the initial estimate f 

with the PSF ftmction for the CCD array that generated the input image for the respective color 
channel. The super-resolution enhanced image generating system can then, for each color channel, 
use the projected image for the respective color channel, and the input images associated with the 
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Other channels to update the initial estimate / so as to provide the final super-resolution enhanced 
image for the respective color channel. 

For clarity, the following description will be of operations that are performed by the super- 
resolution enhanced image generating system to generate a super-resolution enhanced image for one 
of the color channels. It will be appreciated that the super-resolution enhanced image generating 
system can generate a super-resolution enhanced image for each of the color channels in the same 
way. It will further be appreciated that the super-resolution enhanced image generating system can 
perform the operations to generate a super-resolution enhanced image for each color channel 
consecutively, or it can perform the individual steps for the various color channels on an overlapped 
basis. 

After the super-resolution enhanced image generating system has projected the initial 

estimate / generated for a color channel back onto the input image for that color channel, thereby 

to generate a projected estimate image Po for the color channel, it will select one of the other color 

channels whose input image will be used to update the initial estimate / . For example, if the 

super-resolution enhanced image generating system is generating a super-resolution enhanced image 
for the green color channel, it will select one of the red or blue color channel whose associated the 

input image will be used in updating the initial estimate / for the green color channel. After the 

super-resolution enhanced image generating system has updated the initial estimate using the input 
image associated with the color channel that it first selected , it will select the other of the red or blue 

color channel, whose input image will be used in fiuther updating the updated estimate f , thereby 

to generate the final super-resolution enhanced image. For example, if the super-resolution enhanced 

image generating system first selects the red channel, after it has updated the initial estimate / 

using the input image associated with the red channel, it will select the blue channel and use its input 
image to fiirther update the estimate, thereby to generate the final super-resolution enhanced image. 
After the super-resolution enhanced image generating system has selected a color channel 

whose input image will be used to update the initial estimate / , it will generate and initialize an 

array data structure that will be used to receive values defining for the virtual error image v,.. The 
super-resolution enhanced image generating system will initialize the array data structure to have 
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number of array elements organized in rows and columns, the array having the same structure as the 
arrays for the various input images and the super-resolution enhanced image. The super-resolution 
enhanced image generating system initializes the values of the array elements to a predetermined 
number, which, in one embodiment, is selected to be zero. 

After the super-resolution enhanced image generating system has initialized an array data 
structure to be used for the virtual error image v,, it will perform a number of steps to generate an 
error image v for the selected color channel. In that operation, the super-resolution enhanced image 
generating system initially divides both the projected input image p and the input image g for the 
selected color channel into four sub-images p^j and g^j (i=0,l) (here the subscripts are used to identify 
the particular sub-image). In dividing each of the images p and g into a sub-image, the pixel value 
corresponding to the pixel at location (q,r) of the respective image />, g provides the pixel value for 
the pixel at location (x,y) in the respective sub-image p^p gy such that (q,r)=(2x+i,2y+j). 

After the super-resolution enhanced image generating system has generated the various sub- 
images pg and g,y, it will, for each pixel at location (x,y) in the respective sub-image» generate the 

weighted averages Py(x,y), gij(x.y), Pij gij(x,y), p/{x,y),2aid gi/{x,y) inamanner 

described above in connection with equation (7). Thereafter, for each pixel at location (x,y) in the 
respective sub-images py and Cyy the super-resolution enhanced image generating system generates 
the centralized and normalized cross-correlation Cij(x,yJ and determines whether the value of Cy(x,y) 
is greater than the threshold value T, If the super-resolution enhanced image generating system 
determines that the value of c^/x^y) is greater than the threshold value T, it will generate values for 
"a" and "b" as described above in connection with equation (8) and set the value of the element at 
location (2x+i,2y+j) of the data structure established for the error image to 

v(2x + iay + y) = « giji^yy) + * - pA^yy) (i®)- 

It will be appreciated that, in equation (16), "a gi/x,y) + 6" corresponds to the pixel value of the pixel 
at location (x,y) of the virtual input image g as described above in connection with equation (4) and 

so equation (16) corresponds to equation (3) above. 

It will be appreciated that, since, the super-resolution enhanced image generating system 
makes use of equation (16) only if the value of the cross-correlation is greater than the threshold 
value, for those locations for which the value of the cross-correlation is not greater than the threshold 
value, the value in the element of the data structure for established for the error image will remain 
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zero, which is the desired result since, as noted above, if the value of the cross-correlation Cy for 
particular location- (x,y) is less than or equal to the threshold value, the pixel at that location in the 
virtual input image is assigned the value of the pixel of the projected image at that location. 

The super-resolution enhanced image generating system performs the operations described 
above for all of the pixels at locations (x,y) of the all of the respective sub-images and gy to 
populate the array provided for the error image v. Thereafter, the super-resolution enhanced image 
generating system uses the pixel values comprising the array for the error image v to update the 
initial estimate / . In that operation, the super-resolution enhanced image generating system 

initially performs a convolution operation to convolve the error image v with the PSF function. 
Thereafter, the super-resolution enhanced image generating system can multiply the convolved error 
image with a sharpness parameter, and will add the result to the initial estimate / , thereby to 

provide the estimate as updated for the selected color channel. In one embodiment, the value of the 
sharpness parameter is selected to be between "one" and "two," although it will be appreciated that 
other values may be selected. 

After the super-resolution enhanced image generating system has generated the estimate as 
updated for the selected color channel, it can select the other color channel and perfonn the 
operations described above to further update the estimate and provide the final super-resolution 
enhanced image. As noted above, the super-resolution enhanced image generating system can also 
perform the same operations to generate a super-resolution enhanced image for each of the color 
channels. 

FIG. 3 depicts a flow chart outlining operations that are performed by the super-resolution 
enhanced image generating arrangement as described above in connection with performing a 
demosaicing operation. The operations will be apparent to those skilled in the art with reference to 
the above discussion, and so the various steps outlined in the flow chart will not be described in 
detail herein. 

The invention provides a number of advantages. In particular, the invention provides a 
system that generates an super-resolution enhanced image of a scene from a plurality of input images 
of the scene, which input images may have been recorded using two or more image recording 
sensors. The system may be used for a number of applications, including but not limited to the 
demosaicing application described herein. 
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It will be appreciated that a number of changes and modifications may be made to the super- 
resolution enhanced image generating system as described above. For example^ the super-resolution 
enhanced image generating system can be used in connection with a demosaicing operation to 
generate demosaiced images that were provided using image recording sensors for different colors 
than those described above, or that are sensitive to different parts of the electromagnetic spectrum. 
In addition, the super-resolution enhanced image generating system can be used in connection with 
a demosaicing operation to generate demosaiced images that were provided using image recording 
sensors having different pattems of light sensing elements. 

In addition, although the projected image = B^{w^{^ /)) generated using the 

warping operator W^^ followed by the blurring operator B^, it will be appreciated that the projected 
image /?, may be generated using the blurring operator 5,. followed by the warping operator W^, or by 
selected one or more of the respective warping, blurring and/or decimation operators. 

Furthermore, although a methodology has been described for generating the initial estimate / 

for the super-resolution enhanced image, it will be appreciated that any of a number of other 
methodologies may be used. 

In addition, although the demosaicing operation has been described in cormection with a 
CCD sensor array, it will be appreciated that the operation can be performed in connection with any 
sensor arrays, including but not limited to a CMOS array. 

Furthermore, although the super-resolution enhanced image generating system has been 
described as generating a super-resolution enhanced image in the case in which the relation among 
the intensities of the various input images is affme, it will be appreciated that similar operations can 
be performed if the intensities have other relations, including but not limited to logarithmic, 
trigonometric, and so forth, with an appropriate adaptation of the normalized cross-correlation as a 
measure of fit. 

It will be appreciated that a system in accordance with the invention can be constructed in 
whole or in part firom special purpose hardware or a general purpose computer system, or any 
combination thereof, any portion of which may be controlled by a suitable program. Any program 
may in whole or in part comprise part of or be stored on the system in a conventional manner, or it 
may in whole or in part be provided in to the system over a network or other mechanism for 
transferring information in a conventional manner. In addition, it will be appreciated that the system 
may be operated and/or otherwise controlled by means of information provided by an operator using 
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operator input elements (not shown) which may be connected directly to the system or which may 
transfer the information to the system over a network or other mechanism for transferring 
information in a conventional manner. 

The foregoing description has been limited to a specific embodiment of this invention. It will 
be apparent, however, that various variations and modifications may be made to the invention, with 
the attainment of some or all of the advantages of the invention. It is the object of the appended 
claims to cover these and such other variations and modifications as come within the true spirit and 
scope of the invention. 

What is claimed as new and desired to be secured by Letters Patent of the United States is: 



INSOOaO: <wo. 



.03047234Aaj.> 



wo 03/047234 



PCT/IB02/05437 



-18- 
Clatms 

1 . A super-resolution enhanced image generating system for generating a super-resolution-enhanced 
image from an image of a scene, identified as image go^ comprising a base image and at least one 
other image gf, the system comprising: 

A an initial super-resolution enhanced image generator module configured to use the image gg 
to generate a super-resolution enhanced image estimate; 

B. an image projector module configured to selectively use a warping, a blurring and/or a 
decimation operator associated with the image g^ to generate a projected super-resolution 
enhanced image estimate; and 

C. a super-resolution enhanced image estimate update generator module configured to use the 
input image gj and the super-resolution enhanced image estimate to generate an updated 
super-resolution enhanced image estimate. 

2. A super-resolution enhanced image generating system as defined in claim 1, the system being 
configured to generate the super-resolution enhanced image in relation to a plurality of other images 

identified as images } » the system further comprising an image selector module configured 

to select one of the images {g"/ }^._ ^ » the image projector module being configured to selectively use 

the warping, a blurring and/or a decimation operator associated with the image g^ selected by the 
image selector module to generate a projected super-resolution enhanced image estimate, the super- 
resolution enhanced image estimate update generator module being configured to use the selet^ted 
input image and the super-resolution enhanced image estimate to generate an updated super- 
resolution enhanced image. 

3. A super-resolution enhanced image generating system as defined in claim 2 further comprising 
a control module configured to control the image selector module, image projector module, and the 
super-resolution enhanced image update generator to operate in successive iterations using respective 

ones of the images }^._, > each iteration, the image projector module using the updated super- 
resolution enhanced image generated during the previous iteration as the super-resolution enhanced 
image estimate for the respective iteration. 
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4. A super-resolution enhanced image generating system as defined in claim 2 in which the super- 
resolution enhanced image estimate update generator is configured to generate at least one parameter 
in connection with a least squares estimate between the selected input image and the projected super- 
resolution enhanced image estimate. 

5. A super-resolution enhanced image generating system as defined in claim 4 in which the super 
resolution enhanced image estimate update generator is configured to generate the super-resolution 

enhanced image update estimate, for the selected input image {^JJ ^ , in accordance with a virtual 



input image defined by 



where "a" and "b" are parameters associated with respective pixel location (x,y) defined by 



Pi(x>y) 1 



Pi gii^yy) 
gi(x.y) ^ 



and where p^x^y) . gXx.y). p,. /7,^(x,y) , and gf^{x,y) are weighted averages 

around the pixel at coordinates (x,y) in the respective image projected super-resolution enhanced 
image estimate p, or the input image g^. 

6. A super-resolution enhanced image generating system as defined in claim 5 in which the super 
resolution enhanced image estimate update generator is configured to generate the weighted averages 
in accordance with 
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Pi y) = Z Pi (jc + M, :v + v) 

Pigii^^y) = T,A^y'^)Pi{x+u,y+v)g,{x+u,y+v) 
Pi^{x>y) = Z w(w,v) + w,>;+ v))^ 

and so forth, where "w(u,v)" is a weighting kernel and defines a region R around the 

respective pixel at location "(x,y)." 

7. A super-resolution enhanced image generating system as defined in claim 5 in which the super 
resolution enhanced image estimate update generator is fiirther configured to generate the super- 
resolution enhanced image update estimate, for the selected input image | g. in accordance with 

a cross-correlation value c(x,y) generated for respective a pixel at location (x,y) in the respective 
image defined by 



and a selected threshold value. 

8. A method of generating a super-resolution-enhanced image fi-om an image of a scene, identified 

as image gQ^ comprising a base image and at least one other image g^, the system comprising: 

A an initial super-resolution enhanced image generation step of using the image g^ to generate 

a super-resolution enhanced image estimate; 
B. an image projection step of using a warping, a blurring and/or a decimation operator 

associated with the image to generate a projected super-resolution enhanced image 

estimate; and 
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C. a super-resolution enhanced image estimate update generator module configured to use the 
input image and the super-resolution enhanced image estimate to generate an updated 
super-resolution enhanced image estimate. 

9. A method as defined in claim 8, the method being configured to gaierate the super-resolution 
enhanced image in relation to a plurality of other images identified as images }" ^ , the method 

further comprising an image selection step of selecting one of the images {^,}"_ y the image 

projection step including the step of selectively using the warping, a blurring and/or a decimation 
operator associated with the selected image to generate a projected super-resolution enhanced 
image estimate, the super-resolution enhanced image estimate update generation step including the 
step of using use the selected input image and the super-resolution enhanced image estimate to 
generate an updated super-resolution enhanced image. 

10. A method as defined in claim 9 further comprising a step of enabling the image selection step, 
the image projection step, and the super-resolution enhanced image update generation step to be 

performed in successive iterations using respective ones of the images } , in each iteration, 

the image projection step including the step of using the updated super-resolution enhanced image 
generated during the previous iteration as the super-resolution enhanced image estimate for the 
respective iteration. 

1 1 . A super-resolution enhanced image generating method as defined in claim 1 0 in which the super- 
resolution enhanced image estimate update step includes the step of generating at least one parameter 
in connection with a least squares estimate between the selected input image and the projected super- 
resolution enhanced image estimate. 

12. A super-resolution enhanced image generating method as defined in claim 1 1 in which 
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the super resolution enhanced image estimate update generation step includes the step of generating 
the super-resolution enhanced image update estimate, for the selected input image , in 

accordance with a virtual input image g/ defined by 

gi{x,y) = ag,{x,y)+b 



where "a" and "b" are parameters associated with respective pixel location (x,y) defined by 



-I 



Pi gi{^>y) 



and where p^x.y)* g,{xyy), A gt{^»y)> aH^'J')'^** gi^{x>y) are weighted averages 
around the pixel at coordinates (x,y) in the respective image /?, or g,. 

1 3. A super-resolution enhanced image generating method as defined in claim 1 2 in which the super 
resolution enhanced image estimate update genoator is configured to generate the weighted averages 
in accordance with . 

Pi i^'y) = Z ^("' Pi (x+u,y+v) 

-kSu.vik 

Pigi{x,y) = Z w(m, v) p,(x + u,y + v)^,(^ +u,y+v) 
P,^{x,y) = E v))' 



and so forth, where *Vfw,v/' is a weighting kernel and defines a region R around the 

respective pixel at location "(x,y)." 

14. A super-resolution enhanced image generating method as defined in claim 12 in which the super 
resolution enhanced image estimate update generation step includes the step of generating the super- 
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resolution enhanced image update estimate, for the selected input image igY , in accordance with 

a cross-correlation value c(x,y) generated for respective a pixel at location (x,y) in the respective 
image defined by 



and a selected threshold value. 

1 5. A computer program product for use in connection with a computer to provide a super-resolution 
enhanced image generating system for generating a super-resolution-enhanced image from an image 
of a scene, identified as image comprising a base image and at least one other image g„ the 
computer program product comprising a computer-readable medium having encoded thereon:: 

A an initial super-resolution enhanced image generator module configured to enable the 
computer use the image to generate a super-resolution enhanced image estimate; 

B. an image projector module configured to enable the computer to selectively use a warping, 
a blurring and/or a decimation operator associated with the imagery to generate a projected 
super-resolution enhanced image estimate; and 

C. a super-resolution enhanced image estimate update generator module configured to enable 
the computer to use the input image g^ and the super-resolution enhanced image iestimate to 
generate an updated super-resolution enhanced image estimate. 

1 6. A computer program product as defined in claim 15, the system being configured to generate the 
super-resolution enhanced image in relation to a plurality of other images identified as images 

{s^f } 1 » computer program product further comprising an image selector module configured to 

enable the computer to select one of the images {gj.^j , the image projector module being 
configured to enable the computer to selectively use the warping, a blurring and/or a deciniation 
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operator associated with the image selected by the image selector module to generate a projected 
super-resolution enhanced image estimate^ the super-resolution enhanced image estimate update 
generator module being configured to enable the computer to use the selected input image and the 
super-resolution enhanced image estimate to generate an updated super-resolution enhanced image. 

17. A computer program product as defined in claim 2 further comprising a control module 
configured to enable the computer to control processing of the image selector module, image 
projector module, and the super-resolution enhanced image update generator to operate in successive 

iterations using respective ones of the images | g. | ^ , in each iteration, the image proj ector module 

being configured to use the updated super-resolution enhanced image generated during the previous 
iteration as the super-resolution enhanced image estimate for the respective iteration. 

18. A computer program product as defined in claim 16 in which the super resolution enhanced 
image estimate update generator is configured to enable the computer to generate at least one 
parameter in connection with a least squares estimate between the selected input image and the 
projected super-resolution enhanced image estimate. 

19. A computer program product as defined in claim 18 in which the super resolution enhanced 
image estimate update generator is configured to enable the computer to generate the super- 
resolution enhanced image update estimate, for the selected input image [g^Y ^ , in accordance with 

a virtual input image defined by 



where "a" and "b" are parameters associated with respective pixel location (x,y) defined by 

-I r- 



. Pii^yy) 1 



Pi gii^^y) 
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andwhere p^x^y), g,(x,y), piTX^^y), p<^{x,y),aad g^C^.^) are weighted averages 

around the pixel at coordinates (x,y) in the respective image projected super-resolution enhanced 
image estimate /?, or the input image g,. 

20. A computer program product as defined in claim 19 in which the super resolution enhanced 
image estimate update generator is configured to enable the computer to generate the weighted 
averages in accordance with 

Pii^^y) = T^A^,v)Pi{x+u,y+v) 

Pigi(x>y) = X v) Pi{x + u,y + v)g^,(;c + u,y + v) 

. -kiu.yik ' 

• PiK^^y) = Z +«,>;+ v)f 

-ksu.vsk 

and so forth, where "w(u,v)" is a weighting kernel and "(u,v)" defines a region R around the 
respective pixel at location "(x,y)." 

21. A computer program product as defined in claim 19 in which the super resolution enhanced 
image estimate update generator is further configured to enable the computer to generate the super- 
resolution enhanced image update estimate, for the selected input image , , in accordance wifli 

a cross-correlation value c(x.y) generated for respective a pixel at location (x,y) in the respective 
image defined by 

Pigi{x>y)- 'pX'^>y)Ji(x,y) 

im^,y)- {7.(x,y)f){^{x.y)- {l{x,y)f) 
and a selected threshold value. 
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100. SYSTEM RECEIVES BASE INPUT IMAGE AND ONE OR 
MORE OTHER INPUT IMAGES 



101. SYSTEM USES BASE INPUT IMAGE TO GENERATE 
INITIAL SUPER-RESOLUTION ENHANCED IMAGE 
ESTIMATE IMAGE AND ESTABLISHES IT AS CURRENT 
ESTIMATE IMAGE 



I 



1 02. SYSTEM SELECTS ONE OF THE OTHER INPUT 
IMAGES 



103. SYSTEM GENERATES PROJECTED ESTIMATE IMAGE 
USING IMAGING MODEL OF SELECTED INPUT IMAGE 



-NO- 



104. SYSTEM SELECTS PIXEL LOCATION (x,y) 



T 



3 



105. SYSTEM USES EQUATION (6) TO GENERATE VALUE 
OF CROSS-CORRELATION BETWEEN PIXELS IN A 
WINDOW AROUND THE SELECTED PIXEL LOCATION IN 
SELECTED IMAGE AND PROJECTED ESTIMATE IMAGE 



I 



106. SYSTEM DETERMINES WHETHER VALUE OF 
CROSS-CORRELATION IS ABOVE A SELECTED 
THRESHOLD VALUE 

YES 




FIG. 2 



BNSOOCID: <WO 03047234A2.L> 



wo 03/047234 



PCT/m02/05437 



3/7 



0 




107. SYSTEM USES EQUATION (8) TO GENERATE 
PARAMETERS ASSOCIATED WITH LEAST-SQUARES 
ESTIMATE FOR /^FINE SIMILARITY IN THE WINDOW 



I 



108. SYSTEM USES PARAMETERS AND PIXEL VALUE IN 
SELECTED LOCATION IN SELECTED INPUT IMAGE TO 
GENERATE PIXEL VALUE FOR CORRESPONDING PIXEL 
LOCATION (x.y) OF VIRTUAL INPUT IMAGE (REFERENCE 
EQUATION (4)) 



109. SYSTEM SETS VALUE OF CORRESPONDING PIXEL 
LOCATION (x.y) OF VIRTUAL INPUT IMAGE TO PIXEL 
VALUE OF CORRESPONDING PIXEL LOCATION OF 
PROJECTED ESTIMATE IMAGE 



^^^•-NOH 



110. SYSTEM DETERMINES WHETHER IT HAS SELECTED 
ALL PIXEL LOCATIONS 



YES 



111. SYSTEM USES EQUATION (3) TO GENERATE AN 
ERROR IMAGE AS THE DIFFERENCE BETWEEN PIXEL 
VALUES AS BETWEEN CORRESPONDING PIXEL 
LOCATIONS OF PROJECTED IMAGE AND VIRTUAL INPUT 
ylMAGE^^^^^^^^ 




FIG. 2A 
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112. SYSTEM USES ERROR IMAGE AND PROJECTED 
ESTIMATE IMAGE TO UPDATE PROJECTED ESTIMATE 
IMAGE, AND SUBSTITUTES UPDATED PROJECTED 
ESTIMATE IMAGE FOR PROJECTED ESTIMATE IMAGE 
yGENERATED IN STEP 101. 



113. SYSTEM DETERMINES WHETHER IT HAS SELECTED 
ALL OF THE OTHER INPUT IMAGES THAT ARE TO BE 
USED IN GENERATING THE SUPER-RESOLUTION 

ENHANCED IMAGE 



YES 



114. SYSTEM EXITS. RETURNING PROJECTED ESTIMATE 
IMAGE AS SUPER-RESOLUTION ENHANCED IMAGE 



FIG, 2B 
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120. SYSTEM RECEIVES INPUT IMAGES FOR THE 
RESPECTIVE COLOR CHANNELS 



I 



121. SYSTEM SELECTS A COLOR CHANNEL FOR WHICH A 
DEMOSAICED IMAGE IS TO BE GENERATED 



I 



J 



122. SYSTEM GENERATES AN ESTIMATE FOR 
SUPER-RESOLUTION ENHANCED IMAGE FROM THE 
INPUT IMAGE ASSOCIATED WITH THE SELECTED COLOR 
CHANNEL BY INTERPOLATION TO PROVIDE MISSING 
PIXEL VALUES AND CONVOLUTION WITH THE POINT 
SPREAD FUNCTION ("PSF") OF THE IMAGE RECORDING 
SENSOR USED TO RECORD THE INPUT IMAGE 
^ASSOCIATEDWITHTHE^ 



123. SYSTEM SELECTS ONE OF THE OTHER COLOR 
CHANNELS 



I 



124. SYSTEM GENERATES ARRAY DATA STRUCTURE TO 
BE USED FOR ERROR IMAGE FOR PROCESSING IN 
CONNECTION WITH COLOR CHANNEL SELECTED IN STEP | 
123 AN INITIALIZES THE ARRAY ELEMENTS TO ZERO 



125. SYSTEM DIVIDES ESTIMATE IMAGE AND INPUT 
IMAGE ASSOCIATED WITH COLOR CHANNEL SELECTED 
IN STEP 123 INTO FOUR SUB-IMAGES, WHERE PIXEL IN 
LOCATION (x,y) OF SUB-IMAGE (ij) RECEIVES PIXEL 
VALUE ASSOCIATED WITH PIXEL IN LOCATION (q,r) OF 
RESPECTIVE ESTIMATE, INPUT IMAGE. SUCH THAT 
iq.r)=(2x+Uy+j) 



FIG. 3 
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126. SYSTEM SELECTS A CORRESPONDING PAIR (ij) OF 
SUB-IMAGES 



127. SYSTEM SELECTS PIXEL LOCATION (x,y) IN 
SELECTED SUB-IMAGE PAIR (IJ) 



I 



128. SYSTEM USES EQUATION (6) TO GENERATE VALUE 
OF CROSS-CORRELATION BETWEEN PIXELS IN A 
WINDOW AROUND THE SELECTED PIXEL LOCATION IN 
SELECTED IMAGE AND PROJECTED ESTIMATE IMAGE 



129. SYSTEM DETERMINES WHETHER VALUE OF 
CROSS-CORRELATION IS ABOVE A SELECTED 
THRESHOLD VALUE 

^ 



YES 



130. SYSTEM USES EQUATION (8) TO GENERATE 
PARAMETERS ASSOCIATED WITH LEAST-SQUARES 
ESTIMATE FOR AFFINE SIMILARITY IN THE WINDOW 



J-NO-^(^^^ 



131. SYSTEM USES PARAMETERS AND PIXEL VALUES IN 
SELECTED LOCATIONS OF SELECTED SUB-IMAGE PAIR 
(M ) FOR ARRAY ELEMENT (2x+l,2y+j) OF ERROR IMAGE 
ARRAY 




FIG. 3A 
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132. SYSTEM DETERMINES WHETHER IT HAS SELECTED 
ALL OF THE PIXEL LOCATIONS (x.y) IN SELECTED 
SUB-IMAGE PAIR (ij) 

I 

YES 



i 



133. SYSTEM DETERMINES WHETHER IT HAS SELECTED 
ALL SUB-IMAGE PAIRS (IJ) 



YES 



134. SYSTEM USES THE SUPER-RESOLUTION ENHANCED 
IMAGE ESTIMATE GENERATED IN STEP 122, THE ERROR 
IMAGE ARRAY AND THE PSF TO GENERATE AN UPDATED | 
SUPER-RESOLUTION ENHANCED IMAGE ESTIMATE FOR 
THE COLOR CHANNEL SELECTED IN STEP 121. AND 
SUBSTITUTES THE UPDATED IMAGE ESTIMATE AS THE 
IMAGE ESTIMATE 



135. SYSTEM DETERMINES WHETHER IT HAS SELECTED 
ALL OF THE COLOR CHANNELS 



I 

YES 

J. 



136. SYSTEM EXITS, RETURNING ESTIMATE IMAGE AS 
SUPER-RESOLUTION ENHANCED IMAGE 
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